DOCUMENT RESUBE 
ED'126 147 |’, ; 95 | TH 005 403 


. AUTHOR Wilson, Stephen; And Others » 
- TITLE , .. , The Use of Ethnography in Educational Evaluation. : 
INSTITUTION » “Cénter for Ne Schools, Inc., Chicago, Ill. ’ 
» SPONS’ AGENCY, Carnegie Corp. of New York, N.Y.; National Inst. of, & 
Sa Education (DHEW), Washington, D.C. : 
PUB DATE .- Jul 74 ‘ = 
NOTE as: 39p. 


AVAILABLE FROM Center for New Schools, 59 E. Van Buren, Suite 1800, 
a Ghicago, Illinois 60605: ‘ 


EDRS BRICE " 4P=$0,83 HC-$2.06 Plus Postage: 
DESCRIPTORS ° Alternative Schools; *Educational Anthropology; 
us Educational Innovation; Educational Researchers; 
Elementary Sdeondary Education; *Ethnology; 
*Evaluation; Evaluation Methods; Peedback; Formative 
Evaluation; Observation;*Participant Involvement; 
: *Program Evaluation; Research Methodology; *Research 
- - Problems; Student Opinion; Testing Problems 


y) 


ABSTRACT , . Bo ve ahd 
Many people hope that the unique skills and *. 
perspectives of anthropoloyists can help them to maké ‘the — 
contributions to social programs that have eluded ether social ’ 
scientists. Ethnography’ promises to make a significant ‘contribution . 
to evaluation of educational innovations. Before this promise canbe 
‘realized, however, there ‘are Many practical and theoretical problems — i 
: to be solved. An attempt is'made to aid this process by candidly a a 
- sharing and analyzing some experiences in.this ared of applied ~ ; 
anthropology. The use of the methodology, and findings using sy Ro 
ethnography in evaluating am ingovative program of the New. School in 
Cleveland Heights, Ohio’is disclssed.’ Tie evaluation podel used to 
clarify the context in which ethnography is ed is described first. 
The experience of using ethnography is also discussed from /two’ 
focuses: (1) probleps that arose from inherent .mjsmatches between the. |. \' 
sfundamental assumptions and processes of ethnography and those of ae 
evaluation; and (2) problems that arose from the particular ways the 
: evaluation plans were put into practice. (Author/DEP): ' F 
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The Center, for New Schools (CNS), 431 S. Dearbom, © 
; aa 


Chicago, Illinois 60605 is a not for profit organization 
trying to bring about educational change by working with 
S“community’ groups and with people foyihe. to establish 

exemplary learning invivinenta, One part of this work 
is the attempt to develop research and evaluation detivitien 
which’ support the processes of innévation. This attempt ee. 

_has been partially supported by the Carnegie Corporation _ 
and National Institute of Education. 

Dr. Stephen Wilson had mafor responsibility for 
writing this jacer with the assistance of Dre Emile ‘ 
Schéppers, Thomas Wilson, Donald Mo¥re and Phyllis Wilson. 
Jan Aspelin, Betsy Rubin and Neil Gould, all associated 

with New School, exavainnd Heights, Ohio, also made 


a 


significant contributions. 


Researchers responsible for using and conducting 
evaluation studies of social and edugational programs 
' have geown increasingly interested in the use of oe 
"anthropological" methods.’ This interest has arisen | 
out of both a dissatisfaction with’the ,information that 
is produced by traditional evaluation and a recognition of 
the or and importance of the kind of-data that can 
be, gathered by participant observation (i.e., naturalistic- 
qualitative me thods--see Bruyn, 1966; Barker, 1968; CNS, 

" 1974), dur gaveral years experience in conducting eva UB EL On 
studies--mostly on innovat ye educational preg nana heve 
demonstrated that these methods do indeed gather essential 
and valuable information, (See also Smith and Pohland, 1972; 
Ianni and Caesar, 1973; and Wolcott, 1974.) our experience ; 
has also. acquaint d us, however, with many pebblene~cheth. 

/f theory and of practice--which result from mixing ethno- 
“eo graphy and evaluation. 
The prospects for ethnographic evaluation begoming a 
significant tool in the fevelopment of social and educational 
programs depends on the recognition and confrontation of 
tnese problems. By analyzing our experiences in using 


ethnography in evaluation--specifically the evaluation 


fonducted for the "New School" innovative program of the 


. oe, 
Cleveland Heights, Ohio schools--we hope to further this 


process of refining methods for social scientists to contri- 
28 


bute to soc jal change. 
First we describe the evalu tion model to dinbiay the 

* context in which Stead mh vriefly explaining. 
the theoretical Fationale for its use and the specific. 
functions we hoped it would serve in the evaln ion. aan 
we will: analyze the actual experience of using ethpography,. 
from two focuses:. 1. Problems that arose from inherent 
_mismatches between the fupdaniental aeeunpuLone and processes " 
vor ethnography an® those of evaluation and 2. Problens, that 
arose from the particular ways we have had to put oe wae 
ation plans fai practice (ergs, how the evaluation was . 


funded). our illustrations will ‘be drawn from qur experiencés 


in the evaluation of New School. ’ “ee 


. 


I. Description of Evaluation Model, Rationale for Using 


e 


Ethnography 


Evaluation research ist oe as program evaluation 


and, in education, as curriculum evaluation) has grown 


increasingly sophisticated.” Evaluators are growing more q 
anbitione’ in thedr intentions to gather comprehensive infor- 
mareony {For more about these developments, a full disctission 
“of wilon is out of place here, see Bloom, Hastings, and Madaus,. 
1969; Weiss, 1971; Stuffelbeam et al., 1973; Provus, 1972; 
Taylor and Cowley, 4972; any issue of Evaluation Comment. 
The evaluation godel which we focus on grew oft of our 
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” students to identify the school's process and outcome 
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® 
experiences working with alternative schools and was 
developed to be tested in a specific school setting. It 
represents a synthesis of several newer perspectives . in 
evaluation. Although it is inappropriate: to describe the 
model in Aptats in this paper (see CNS, forthcoming, for this 
description), it is necessary to outline some of its essential 
characteristics tp explain the context -in which the - 
ethnographer- -evaluator worked, We will briefly déscribe 
the concrete activities of the New School evaluation and then ef 


consider the-follbwing principles and rationales underlying 


the use of ethnography in valuation--formative evaluation, hes 


,process analygis-goal free evaluation, and assessment of 


subtle outcomes. 


To realize the goals of evaluation (goals which will 


-be detailed in later sections), Cis undertook the following 


activities at New School. As an outside evaluation team, 


we worked with an ‘evaluAtion committee of teachers and ~ 


goals and to learn what the committee thought the focus of 
the evaiuation should be. Additional focuses were: derived 
from interviews with administratgrs and staff not on the 
committee, student suggestions during an all-school meeting, 
inspection of documents eadevent to the innovations being 


studied (e.g., original proposals, teacher memoes), obser- 


vations made during an initial on-site visit, and GNS experi- 
p / ss ‘ : ; ? ” 


ences in similar settings. 

Data was collected by multiple methods (interviews, 
questidnnaires, observation). Participants were involved 
+o the design and activity of data ‘collection wherever 
poss ible. CNS trained a sélected group of participants in 
intefviewing and data analysis. Two interview sequences 
“ware planned, but the second was not completed becausd of 
scheduling. difficylties within the school. 

CNS worked with school participants: (through regular 
consultations with’ the eyaluation committee, a two-day 
workshop during the school year which involved the whole 
school population, and presence at a significant year-end - 
planning meeting) to help them digest @Valuation information 
ee work toward their goals. CNS also ‘prepared a final 
Pearle ea report analyzing events related to the working: 
of the innovations? in the school. 

X part-time, student participant observer (trained by. 
the CNS evaluation team) Joined the school commmity® from 
the beginning of the evaluation. The use of. ethnography 
was essential in the accomplishment of the svetoalleweale 
of the evaluation--i.e., formative feedback, providing a kind 
of information helpful to: participants in reaching ;their goals, 
analysi’ of what happened in a way that would be useful both - 
to _participants and té fits adaptors of the innovation,” and « 


obtaining information about subtle changes in student behavior. 


fs 
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In' the ‘aaa 5 pages wé elaborate on these goals-- 
k 


donation, ‘feedbac process analyis and goal free evaluation, 


assessment of aubtie or unintended outcomes. 


: bs) e 
l. Formative eyaluntion « relevance to the reality of participants 
“Formative evaluation te the attempt to use feedback 

derived from the evaluation to help shape’ on-going prograns 
and to help partictinte esaah their goals. Evaluators: have 

a responsibility to the people in ihe eaudational institutions 
being studied to help them to be successful in their endeavors. 
The people in the settings have a right to feel that the ; 
evaluation is theirs and that they can help to decide on the 
focuses of the evaluation. In fact, they should feel like ° 
co-evaluators, ‘The evaluators fulfill this obligation by 

working with ine people in the setting to help them to define 
their goals and expectations in relation to the innovation, 
by gathering ‘ugeueians relevant to the way the imovation 
works, by ‘feeding this information nach to the people in the 

; schools, and by helping the partivipants to divest the infor- 

mation and change ‘their organization. Furthermore, the 

‘evaluators’ links with, the developers of the innovations 

and with other schools who have implemented ‘similar inno- . 

: vations, nakenthien a good source of assistance related ‘to’ 


the participant's goals. 
* 


af 


Hiioeraphy is particularly well suited to hase 
formative functions of the evaluator since learning the 
frameworks by which the participants: ina setting interpret 
events is a fundamental task of the participant observer. | 
Many of the prdécedures used by antifropologists are Jaataned 
to help the researchers temporarily suspend their ow perspec- 
tives and to understand the daily life’ of the setting they 
are etudyAng in oe same way the participants do. 


Practitioners in schools often complain that evatantore 


and researchers who work with schools do-not understand the 


Erne of schools and classrooms. As a result, many ° 


the suggestions derived from research and from -standard 
evaluation design are dismissed as being irre evant to 
educational realities. The data gathered by ethnographic 
evaluation attempts to avoid this shortcoming since acquain- 
‘tance with the daily realities of the participants is 
intrinsic to ethnography. 

The formative evaluation that we have been describing 
is a relatively new and increasingly popular aspect of’ 
evaluation. Frequently formative evaluation plans have gv 

paid little attention to how the information would be used. 
Planners have assumed that data related to- program goals: 
will have obvious usefulness, and their responsibility : 


ended with the collection and communication of information. 
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Those who have studiéd organizations and the process’ 
of planned change (Havelock, 1969; Bennis, Benne and Chin, 
1969; Sarason, 1971; Shmuck, 1972; Lortie, 1969) realize. 


that this kind of information not automatically trans- 


lated into action. , In ae ve an impact, facilitators 
a 


\ . 
of-.change must know as much as [possibte about the structure 
of an organization and offer information in the right form’ 
to the right people at the right time. 
gent peop & i 


$3 "Participant observation is a potentially valuable 
source of this analytital informatian about the structure 
wa dynamics of organizations. This understanding can be 
used,to guide the evaluators in their attempts to use the 
information gathered to help the participants reach th@ir 
goals. Furhtermore, as the evaluation team sib action to 
help accomplish the desired changes, participant observers 
can monitor the effectivengss of these attempts, waite ate 


successes and failures, and make suggestions for more. 


effective strategies in the future. 


ES Process analysis, foal- free evaluation 


Evaluations of educational programs are usually designed 

. 4 : € £ P 
in a very limited fashion--oriented toward giving information 
only about the specific setting being studied. It makes 


more serise, sini Sac think of evaluation ‘as an integral 
ne? hgh é 


’ 
~ 


, 


V, 


: part ‘of the chain of processes relating to the qpvenan= | 
“ment, dissemination and utilization of educational 
innovations. This means that each evaluation ue avor 

‘should povias comprehensiwe information helpful o those , 
‘who may try. to implement the innovation in other settings-- -_ 
i.e., information that helps them decide how this oa oie 8 

‘vation is appropriate to their circumstances or wha 
Kinds of chankes will have to be made. This information ‘ e 
should include data about,the day- to-day functioning of -— 
the innovative program--2.8., how it was actually { Lenented,: 


how it accomplished what it did, and why it didn't f Tilt. . 
‘other expectations. Because: of its flexibility ahd 4 i \ 7 aa 


af 


‘comprehensiveness, participant’ observation has long been! 

used to stully such detailed ~social processes in communitiys aN 

(Vidich and See 1958) and in organizations (Scott, 1965). 
® The _open- “endtanéss and theoretheat flexibility of ie 

evineeeachy also equips it to address some cies concems 

of evaluators. Stake (1967), Scriven 7), Eisner (1967). 

and Provus (1967) ee ee traditional evaluation 

avpnbeches which prematurely 1imit the kind of information 

collected. Stake has sugeated that evaluators. aust try 

to look at costs and unintended outcomes as well as assessing 

intended outcomes. Similarly, Scriven has suggested that 

evam@mators take a "goal free” approach in order to be sensi- 


: @ ' 
tive to important information that might be outside the 
2 . : : 


, . 


\ 


<a 
‘ 


original, plans of the program. Eisner points ot et a 


narrow foéus. on objectives often hampers underdtanding of 


‘educational programs. Provus sees evaluation as serving 


functions outside the immediate program being evaluated , 


‘and hence sees‘the need for collecting information about 


many aspects of the program, This comprehensive informa- 
tion is also pase tigt in serving the formative function. : 
The’ nature of! innovation in complex’ social systems 


means that there gre likely ‘to be consequences and impli- 


_ cations that are difficult to anticipate beforehand. Since 


they cannot be anticipated, ‘it iis impossible. to plan to 


measure them as is requited in standard experimental. design. 


Yet this unanticipated effects ‘are important to assess, 


both for the purpose of. making comprehensive judgements 
about new programs and for providing information necessary 


to thosegtho seek to implement, the innovation in other 


oe) . 


settings. ; 7 

Ethnography has long’ been concerned with tie. ceotien 
of not knowing the relevant variables before entering a 
setting. - As a consequence, quntdtative vadearchens have 


developed systematic inductive methods of letting. their 


-. concepts es theory emerge from on-going. gic de with the 


setting. "(See Glaser ‘and Strauss, 1967; ‘Bruyn, 1966; Smith, 
1971.). These methods, known as qualitative, grounded research, 
eo oe $ 


12 
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‘are thus especially appropriate For ‘use by evaluators who 


‘i 


have criticized the narrow focuses of traditional evaluators. 


\ 


“3. subtle SuRCOMEE Ss airtioult or impossible to measure 
; Evaluative studies. typically rely on measurement by 
ae “taate and other psychometric instruments for Stee LON 
‘about achievement, ‘of desired outcomes in: studied behaviors. 
“There, has eet, however, a géoning discontent with these 
‘instruments, Thewe are many methodological problems for ze 
their use in evaluation (e.g., timing of test administration, 
_desjen Siarecbeniatine of the instruments, see, Porter, 1973). 
Mgny practitioners feel that behavior in these kinds of 
specialized data collection situations (tests, questionnaires, or 
etc.) is not closely related to usual daily behavior (see 
Jackson, 1968, on tedchet\distrust of tests). Much social 
science research has similarly pointed out the artifact 
produced ‘by standard tedhniques of social psychological 
“research (Deutscher, . 1965; Rosenthal and Rosnow, 1966). 
Naturlaistic observation (Willems and Rauch, 1969; Barker, ‘ 
1968). and unobtrusive measurement are increasingly seen as 
important tools for social scientists and evaluators to use 
when They want: information about chan in student behavior 
in everyday. contexts rather than information about performance te 


~ in specified measurement situations. Ke ° : ‘ 


7 & fs : 
* P ‘ ‘ Pi 
. _ ? iss ’ 


“The nature of the oujtcorie goals of many hae 


educational ‘programs also raises doubts about 


f 


of traditional measurements. 
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‘ 


‘Many ‘of (UnEee new programs 


be % 


adequacy a 


aim to bring about subtle changes in higher. levél cognitive 


and affective student behavior--e erg., critical thinking, 
. t 


sense of self-worth, attitude towarg Sqpool, vocational 


orientation, ethnic identification, etc: 
: 


‘ 


Psychometricians 


admit ‘that their instruments’ are weak in assessing these 
= 


; kinds of qualities. 


» 


For the Sake of convenience, rather than Seyi to 


assess these difficult-to-measure outeomes, traditional 


evaluators use whatever devices are available--e.g.,.stan- 


dardized tests. This tendency often results in an emphasis 
a Wy - . 


; in evaluation that is not reflective of the true emphasis of 


the programs being studied. 


Evaluators have a ae ae ae 


to find. “ways to provide antennas ton even about these elugive 
H 


outcomes. < 


bad 


4. 


ea ee. not used for these purposes, partici- 


pant obse fation offers hope of resolving some of these 


_ problems in assessment. Field observers, because. of their’ 
rs « . * * 
contol wide ranging, relatively unobtrusive presence © ‘+. 


"in a setting can obtain information about day-to-day behavior 


. 


instead of only Speetalines test behavior. 


customary research Procedures 0 of this iid ig Pe 1 


A 


Furthermore, “the 


’ 


> i a iz 


"at crucial events andvsensitivity to a multitude of behavior 


ic caamelibiad la possibility for evaluators to collect data 
“about higher leva “gognitive behavior and affective ovlent 


taedonay | ‘Moreover, participant observation data'can dlso 


be used as a ee basia on which td design 


quantitative approaches. 
a 
Potential ‘ , ra 
This section has presented several rationales. for 


including' anthropological techniques in evaluation studies. 


There are many fortunate fits between qualities intrinsic 


& 2 


. ? . : 
to the ethnographic research process and ThE: DULPERES and 


needs of evaluati6n. In spite of this great potential, 
however) ethnography has rarely” ‘been used for these purposes. 
Much more’ development is needed before ethnography can 
fulfill its promise in these contexts. , 


- . = 


~~ 


‘II. Realities. of using i a a in ereManion” 


Our purpose in this article is to ee Fa some areas 


in which this development must occur. Our expériences 


“conducting ethnographically oriented evaluations of inno- 


vative eflucational programs have alerted us to many 


3 difficulties. © Some seem -to arise ‘from inherent conflict 


. 


between the processes of sea processes of 


* 


- 


particip t observation; eenees arjise_out of the particular 
choiées we nde had to yates in putting ha evaluation plans 
Into Sranitee, (Note that these latter difficulties? while 
not As inevitable as the former, Pat nonetheless confront 

_any Bvaivatars faced with the same financial, temporay, 


preerere s contraints. as‘we were.) 
{ 


Inherent difficultias | 


i” pa were | — vk oe 


t & 
Social sci tists have always experienced difficulty 


‘with the ethical questions of their’ responsibilities td 
"+ the people they study. day, many organizations and 

) : 
", communities no Langer welcome researchers and evaluators — 


% ur legs they can demonstrate that, their work will result in 


s 


z direct benefits to the pommunity Being studied. The intensity 


~ 


and intimacy of thesparticipant observer" Ss interaction with 
the SOMME Tse hé studies moss shat he is confronted more 


directly with these ethical demands than are other- Kinds fo) 


ad 
restarchers.. Harthexmers, the insehsitivity of participant 


> 


observers: in the past--ise./ those. who have t&ken a Ycolonial” 
: co a 3 ‘ <i ” : 

stance toward other cultures and toward, minority cultures 

within our own society--has made 'this kind of research an 


especially popular target of criticism. ’ : 


¢ harkhropetontcts have “sought ways ‘to be diseful to the 
people they study, to’ serve the communities which for. so 

long were their subjects (Wax, 1974). *Formative evaluation 
. seems an especially “appropriate way toe of service. 


7 aie y 
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Our experiences have demonstrated that’ this marriage of 


ethnography and feedback is not easily accomplished. 


. Certain requirements of successful ethnography work against 


the Formative ia function. 

The richness of participant observation as a source 
of information derives in. Wart from ‘the esa of 
the field researcher. In sie obhervat ions the 


researcher works to be- accepted in a special liminal role. 


He avoids being identified with any particular group within 


the’ community or with any PALtIGULSS. position on important . 


‘., 


i issues, ‘This non- advocacy is seeunbte so that participants 


‘“from all seetors of whe community and with ie aa of 
a 


views will feel, that “they git a sympathetic r and are 


wiivine to be SPLCERERTS and respondents. : 
Inevitably, in Pee ethnographic studies, the nature 


of social interaction means that the researcher cannot 


_veally avoid making some comments that offend or distance 


some a dani aac His special role of non-advocate, 
however, “leads participants to be willing to discount the 
occasional distancing comment. ' 

The ‘special combined role of formatfve evaluator and 
ethnographer changes ‘this. The researcher wants to provide 


information useful to the community, and the comniind ty 


expect to receive this Ant OMRON The evaluator- 


a 


a aie 5 rapes a dilemma ie t suf" e hnographers do 
at? 


» 


oy and sch 
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not. His dual role requires him to do things et ped 
most likely limit future information. collection in signif- a 
icant’ Ways. 


- & 
To ilaustrate, we quote excerpts from three papers 


written by the field worker, a teacher ae a student at New 


School. The field researcher was aware of his difficult 


e 


dual role. , = . é es Ys . 


(Field. worker) ‘There was a tendency for eureeek 

the community to have the settle disputes on some 

issues which were emotionally chiewed,” One example 
of this took place at a confrontation between four 
" students ‘and a teacher. The students contended that 


they should be ete complete control of all New 


A School ; co ‘waking, including curriculum selection 


policies. a . ; ‘. 
The Seen onan eey that because ‘of state require- 
ments for curriculum and the school board's authority 
reeataine policy making, the students were not in a a - 
position to be in eomriate control of these things. 
tha. Feuabes bun wilco. both-eides ot the argument 
until one of the students turned to me and asked, "You've 
been observing us, don't you think the students of New | 
_ School could handle. these decisions?" It me obvious 


that I could not support either side without eer 


. 


1 \ . . 
. 


. 


dizing my effectiveness in future interviews, Equally 
giv iola-uae the unreasonabl osition of the students 

“in light of the facts. My. ‘reply both praised -the 
avidity of New School students to function as a 


decision-making body and emphasized the circumstances ey 
. 4% ‘ 


~ which woyld make their proposition impractical. 


. 
a ‘ 


ii Some of the participants realized the difficulties ¥ 

ee . for the gnthropologisé, but heey had, engaged the evaluation 
“8 ‘team with the expectation of feedback. They wanted to 
impfove their cuoekan and were eager to get information. - 
“Thus they were unsatisfied with the, reticence of the 


t ¥ 


* a! 
participant observer. 


(haacher at New School) No doubt the question of — ! 
active role is BPE ENARE of a dilgmma _ none aaa , 
_ Pant observer, but I think it is possible for an ; 
. : anthropologist to actually maintain dbieotive ae baneds* 
. without appearing to do soa. This* may be a rare ability. 
, ad : Regarding feedbacks this may also be a Ailemma. fer 
an aitheopeloetet, since anthropologists are not 
especially interested in effecting thange in the % 
community they are eehins with. However, since 
‘positive change is he point of the while evaluation, 
& program, as far as ‘the New School community is Goricemed,’ | 
. I think that continual, detaitea feedback is important. | 


‘M has not given us enough of this. ie eo 


! ‘ : * 


0 * 
i9 


tte 


¢ . 
Some students were even more adamant in their com- 


ey : ew P 
plaints about the lack of feedback. -Their judgements 


/ 


/about participant observation were made primarily in 


| terms of amount of feedback. 


ry 


“« Little value to us. | 


e 


(Student) The majority of the students felt that 
2 , \e 


M has not done an adequate job as our participant 
observer. Although whl of them were willing to’ 
entertain the idea that he was being helpful to 
the outside-evaluation team in their, part of the 


' ; my 
_ evaluation, it was felt that he was not being aNectiy 


helpful to us. 

Even the Mapex ere who ce1t/ that M was doing a 
. good joo, because they saw him around, going to 
classes, etc., and he was always taking notes on 
what was going on and what people said,’ had not gotten 
any feedback from him. Most of them were nies endaeh 
to’'feel that this was due to the fact that they had 
never really asked him for any. (The atidénte who had 
asked him for general feedback, however, felt that 
their question was never really gnewered. (The ut 
tithe wee thought by these students to.be true of the \ 
evaluation in general, sdpbutally the evaluation work- 
shop.) For the most part, people seem either not to — 
know whatM is doing--an interesting conment in itself, 
or to feel that the job he heer been doing is of very 


. 


, 
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The expectation of feedback is a new-element that 
‘must be’ integrated into the role of the eT DARE CapnE oe 
evaluator. Other individuals*can try to take over the = g \ 
responsibility for converting the information into feed- . 
back (as is described later), but» the commun iy continues 
“to. link the ethnographer with returning information. 
Judgements ia ‘evaluation are often made in terms of how : , 
useful and AoW speedy the feedback is. a. 
2. Sacred cows. ‘e Ls 
. Traditionally one of the outstanding strengths of 
’, participant opeervation asa i has been its 
ability to help researchers to aes facts about social e 
interaction in communities, facts that are usually hidden « , ; 
even from t participants. The ethnographer ‘strives to. . 
look at. events without taking anything for granted,’ and 
he tries not to be locked TT any particular perspective. ; 
These qualities of: participant observation can contri- 
bute icone to evaluation. . The non-accomplishment 
of program goals can often be traced, = what Smith (1974) 
has called latent eeipewa~outackiuwieaced organizational oe 
patterns of behavior. 7 the same time that people want to 
know what is _frustrgt ing ‘their ambitions, they may also want 


to avoid this informations Survival | -in an organization often « 


Ae me 
r 


~ 
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leads participants to minimize their awareness: 
Participant observers des ina unique position to be 

useful by becoming a "conscience" for the community. They 

cultivate an awareness of ont Sympathy with the goals and 


mores of the people they work with so they. are not. strangers 


to the eating Sbanewsbiee of the participants, At the same 


“time, however, they maintain an essent¥@l distance from thes 
goals ahd mores. They remain forever partial outsiders. 
As. a result, they can point to sacred cows and to bee about 
the community that participants are reluctant to ackk ze. 
The participant observer in his evaluation search to : 
‘identify which elements of the program support or inhibit 
realization of desired goals has a high probability of 
identifying these blotted out-or never revealed facts. 
For some issues he may identify particular persons or groups 
that -work against desired goals. The nature of these dis- 
coveries poses several dilemmas for an ethnographer-evaluator 
who seeks to give formative feedback to the community: He must 
carefully weigh the canons of confidentiality agains the 
community's need for information. Participants may feel a 
resentment that the field: researcher, who shares so much 
of the life of the participants and seems’ so sympathetic, 
should commit the saeniians of questioning the sacred cows. 
At the same time that they have been open and "progressive" 


enough to see -the value of an objective viewpoint, they may 
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mak ing-;all teachers and students having a voice-- a small 


‘had no one to blame but themselves for their lack of parti- . 


? 4 


also be defensive about what..-the observer sees. These 
negative reactions can lead to wignelding information from 
the observer, a hostility toward the person, a discounting 
of his contributions and an inability tp use his ‘feedback, 
We can illustrate by describing such events at New 
School. The field worker identified a phenomenon which 


some students called the “power clique.” In this alter- 


native school whieh cla&imed.to want democratic decison 


group, of students and teachers seemed to be actually 
: Fad e 


dominating the decisions. Other teachers and students 
he 


resented the fact and complained to each other and to sna lt! 
observer, but could not~or did fot oreunion to express their , 
feelings: The "power clique" was aware that they were making . 
many of the decisions, but they felt that everyone had an © . Ae 
opportunity to participate since all meetings and announce- eae 
ments were public. They insjsted that those who did not _ és 
participate were unwilling as in the desired work and 


. 
. ° 
sf ‘oe 

5d t 


cipation. 
The decision making process in alternative schools is _ 
too complex to discuse completely here. ‘(For more details, 
see CNS, 1972, 1972b; Wilson, 1972.) There are grains of 
truth in both perspectives represented in the example. The 


2 . 2 4 . P 
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° field worker on “this project felt jhat- the “power clique" 


ae _ phenomenon, nad ramifications vdjond decision making--9.g., 


- 


influencing how various studénts’ beeame involved in the 
léarning Progra at fhe @ Achool. gare an to communicate 
* the feelings of the n hee ee and to alert the power: 
chique to ne’ saci aSAAWEGIE in which they were contributing 
_ to the non-participation of others. > 
‘The participant observer learned early in the evaluation 
‘ that he was being “identified with the power clique (who also 
, + formed the sch8ol's. veachen-etudent, vhluation committee) and 


that this association was limiting his access “to information. 
x 


. 2 
7] « 


(Field worker's report The specifig incident which 
focused attention on my association with the committee 
; saute what I sie! questioning a student after class. ; 

“I had overheard this ‘student complainisg with two 
‘others prior to the class about the “haphazard presen- ‘ 
tation of’ the subject (material) "by the student-leader 

; vot fhe c course. When I questioned the stydent~efter the 

. class, she said that she was happy with the course and 
that ‘she sould + think of any improvements for the ota 

F selsions. - Not wanting bg embarass her dian" 
confront) her with her, cariie® statement. 


> 
Tae Siac weeks” later it bécame apparent that- the Yeason 


some students were reluctant “to be completely canaid 


7 
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with me was that I was asspciated with what ee ee 


the "power cliques" ~~ 
* . : 


“The observer felt that it was essential to lea 
cate this information to the power etique for the /sake of 
improvihg the program... He set out to discover w ich sthdents 
were not showing uF at school and to contact them. He sient 
time with the teacheee who were not part of the active group. 
Although the~power clique acknowledged the existence, of 

~~ uninvoived ‘people, and. claimed to want to kn¢w more about . 
why they were uninvolved, x08 observer “fel that they were 
not as aware of the issue as they might | ot ‘As he tried to 
emphasize the non-involved, many members of the power clique 


4 
grew dissatisfied with his activities.” . 


4 
3. Not a miracle worker “ 
The rationale proposes that a field worker ‘can gather “': 

information that is useful to participants in helping them 
adeonpitch their goals. The evaluation team uses this 

» information na help the school "solve" its problems. 
le ‘discovered that there is ad ger that paring 
ae cae much from the ene in the formative 
yevaluator Bolé.” Because he teeplwaye around a people see 
him constantly asking questions and taking notes, they assume 


— 


= 


that he has a thorough understanding of the community, that he 
can. therefore solve their problems. There is a‘large likeli- 


hood of disappointment, 7 - 
‘ : es , . ; ' 


(Teacher) I think that a participant observer with 
"training in anthropology, psychology, and education oe Ree 
can‘be a valuable part of the evaluation” of the 

‘ program. However, I wonder if the. information that 


-he can supply is all that important and useful to 


the New School community itself in terms of the 
program's improvement. 
(Student) Some {students| felt that there was a— 
( “need for him [observgr] to be kmowledgable in the ‘ 
fields of education, psychology and possibly anthro- 


pology. ° 
\ its a f ss 
"Being knowledgable in education, -psychology, and 
eae vende meant skill in. helpine community 
members solve their prob ems; In fact, much of their 
Sie in this kind of evaluation was the help it seemed 


to promise in working toward their goals, and they were 


disappointed in the field researcher's ability to help them. *- 
ee ; 


Social intervention based on traditional-research has 


- been largely disappointing and criticg are wondering just si 
; hae "a , Soe 


~ilte 


, 
4 


what contribution social science can make. Unfortunately, 


holistic -Anowledge of a community or organization does not 
automatically translate into appropriate actjons to be 
‘taken. ‘Even: stiview bie: organizations may quickly eyolve’: 

- rigid structures which keep feedback information from having 
an effect. The ethnographer-evaluator is not.a miracle 


s 


_ worker, ‘ 


4, ‘one kinds of information are harder to" get than | others 
Psychologists are ‘exploring. the use’ “OF nétupalteti 
observation. to study behavioral ‘dispositions such as 
attitudes and cognitive abilities. As explained in the : 
‘section on rationale, many . ‘people doubt the “éndTtivity and 
‘Peoneresnabitiey of results drawn froin paper: and pencil 
instruments. They seek other, more comprehensive, methods 
of assessing both student abilities and affective orientations. ° 
-- We discovered. that participant. suniation was indeed 
a powerful technique for gathering reliable and. valid nfor- 
mation about: student behavior changes. We also aiscoyered, 
“however, . that pre a “amount ¢ of energy” required "to ‘gather ‘this 
; Anpormanetae aia precluded doing anything else. While a 
participant observer may ‘be sin: to simultaneously gather | 
information about a wide Narteny of PORTE ae must also Bs 


o 
dee ene on priorities, He yale be able to caliisat less — 


- 2 ‘ 
: * 
.. 
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complete information about low priority is@ges. This _ 


problem of priorities is aepee ey severe hen: choosing ; 
ad between gathering data on ‘more, manifest fomuses such as 
. > social interaction and more subtle, less overtly expressed 
focuses such as cognitive skills. A focus on the latter 
would leave little time .for the former. . 
The wisuation tean, field observer, and SOBEHSy 
student evaluation gomittee of New School met early ‘in’ 
. the evaluation to decide what issues the investigation should 
focus on. ‘Two of. the issues dalected were student learning 
(e.g., problem solving skills) and formation of school’ 
community, -sense of belonging. The. following observation 


illustrates the dilemma in choosing priorities in the field. 


Ses, ‘ s. 


- . 
- The' observer was watching a science class. Students 


were proposing experiments they 'were going to do and 
soliciting help from fellow students and the teacher. 
Who interacted with whom aid in what ways was signifi- 
— reer Bchool*s "goat-of-buttaing-2——————— 
-close, supportive community. What kinds of experi- 
ments.were proposed, the reactions others offered, and 
the counter reactions of students were significant 
data shout student learning and problen solving skills... . 


The observer could not focus his attention on all 
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“N 
these eventd at the same time. Similarly, when he 
formulated the questions he wanted to informally ask 
the students or teacher after class,. he could not ask 
about every issue he was interested in--e.£y5 why a 
certain student asked for help from dae Btiitente and 
not from others or what a student thoughtewhen the 
touche made the comment she did about a student proposed 


Ethnography in evaluation can give information about 


all kinds of issues. Those who seek to use it,, however, 


will need to carefully examine the limits. 


“i Can ven study yourself? 
In addition to helping the tieveatten team design 
‘effective .ways of presenting information to the school, it 
seems that the participant observer might be in an ideal 
‘ position to evaluate the evaluators. © He could obsefve how 
- the school community responded “to the-activities-set-up—by———___ 
” the evaluation team to help the community use the feedback 
and hew the community actually-used or failed to use the 
information. He could make suggestions for subsequent data 
collection and intervention activities. ; 


| discovered that a participant observer cannot easily 
fulfill this role if he is also fulfilling all the others, 


a P 
* . * e 
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* The participants tend to identify the observer with the 


evaluation team and may not be as candid with hin about their 
. \ 


reactions to the evaluation as they are about other matters. 


% me fo 
B. Practical difficulties : = 


Wolc&gpt (1974) has outlined several criteria for 
éd - . 
excellence, in the ideal STRHOEEAyEAS situation. In this 
section we will briefly outline some of the practical limits 


and constraints that we found to be a part of using ethno- 
: ) f 
ce 


CJ 


graphy in evaluation. 
1. Assistence organizations with specialized skills, 
such as CNS, often Hind tnaumelves asked to ‘conduc evalua- 
¢tena in distant cities. Since contact between the evaluation 
team, field worker, and school coninunity is an important ark 
of the model, we found that limited contact--by mail, telephone 
and periodic visits--is insufficient. As the only member of 
the evaluation team actually present in the field, /the 
_-_researcher.was subjected to many pressures and drains on 
his time-+e.g.,,becoming public relations man for the whole 
__project. ‘And, over a long distance, it was difficult to 
co-ordinate the on-going use of participant observation data 
to shape the focus of the other aspects of the evaluation 
(e.g., tests, interviews) or for data eherging from the tests 
and interviews to be used by the observer as a focus for his 


* 


ed 


field work. 
ge. 2. the greatest appeal of the model to school practi- 
tioners has been its promise of immediately useful feedback,, . 
and members of the school community asked the researcher to 
\ switch congtantly from observer ta participant. We had 
hoped to f lter feedback primarily through the evaluation 
team, but the community wanted more continuous Sesdiacks 
Our experiences suggest that at least two people are required 
in the field--the observer and an advisor-feedback specialist. - am 
The demands that are made on a single ethnographer-evatuyior 
co the field can seriously impair his r. eee and ‘the 
strength of his low key participant-observer role, ; 
3. The ecoriomic realities of money available for school 
{ " -evaluation often severely limits the “ideal plans. Ideally, ® 
as Wolcott points out, ethnographers would have a long time 
in the field to-build relationships, understand the community, ~ 
develop tentative: hypotheses and test emerging theories. They ‘ 
would ideally have a long time after nape work to analyze 
——*-—what-they-had-seens—Many-of-the-schools to~be-studled-ao————————— 
not have es money to support a field worker for the 
ideal length of time.— Moreover, they are.so eager for the.— 
feedback they need that they cannot allow an ethnographer 
-sufficient time to settle into the community or to analyze ‘ 
fully what has been observed. (See Lundin, Nelson and Gianotta, 
1974 for another description of these different time frames.) 


=80 
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We attempted a compromise by using part-time observers 
for a single school year. Although this arrangement made 
unusual ‘and difficult demands on the participant observer, 
we felt it better to have a continual and regular presence 
in the eonmuntty rather than the fragmented spot visits that 
are typical of evaluation. 
ot We asked the part-time observer to share -his tentative 
thoughta with the community rather than witholding and 
testing hisAdeas. Still, this compromise caused problems. 3 
The school members felt they didn't see enough of the observer 
and he, in turn, felt that he didn't have enough time to do 
all that was called for in the model. The lack of funds_ 
also places limits on the skills and experience of the e \ 
observer-evaluator a school can afford to hire. a 

Anyone working with similar time and funding limifa- 
tions must understand thesé practical limits. An evaluation see 
plan should either supply sufficient funds to support the 
‘observer over time or it should limit the focuses and 
functions that th&.observer ig, to serve. 

4, Critics have become increasingly vocal in‘their 


demands that social scientists find ways to-be responsive 
and’ useful to the people they study, especially anthropologists = 
who work with the communities so intimately. Anthropologists~ 


who wish to be responsive to the communities under study 
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ecw 
will face some conflict of values--that is, the community.— 
= . does not necessarily accept the. rules, costs, or values 
of social deience. This potential tension is aepecialiy 
great with ethnography, for the field researcher may appear 
not to be doing much that is different from other community > 
members--he talks to people, attends classes, asks questions. 
Community members observing this kind of research-~ e 
especially when the community itself is paying for the 
evaluation--may wonder if their money is being well spent. 
Teachers and students at New School were aware that their 
money could have been spent on other priorities and they 
wished to see an immediate return. indoubtediy, “many” 
® ethnographers in the future will face this strange situation, 
being the employee of the people they are studying. a : 
All of these very practical constraints and limitations 
must be taken into suhetaaretion When planning to use ethno-. 


graphy in evaluation prograns. . 


III. Summary ; 
Many people hope that the unique skills and perspectives 


of anthropologists can help them to make the contributions to 
d social programs that have eluded other social scientists. 
Ethnography promises to make 'a significant contribution to 
* evaluation of educational innovations, Retfor itisiprqmtomise 
can be realized, however, there are many practical and — 2 ee OS 
theorettical problems to be solved, In tttiks centile we, theme 
attempted tto aid this process by candidly sharing and analyzing 
our experiences in this area of applied anthropology. . 
330 : 


: Anthropologists, of course, use a large variety of , - 
\ methods, including-the ‘tenty,, intenviod and questionnaires 
traditionally used by evaluators. ienthrepoloztoal™ used in 
these contexts, however, refers to participant ‘observation 
and field interviewing--methods long considered the main- 
Stay of anthropology. In this rapes “anthropological” ‘4 
and “ethnographic” will “be used interchangably to refer to 


the activities of the participant observer. 


2 We concentrate Desay on evalvation in educational 1 
settings, ai though ‘ont of our Pieneeyae would apply equally 
well to other settings. 
‘3 The New School evaluation proRean Wie shaped both by - 
the theoretical rationale and by practical constraints. 
For example, with CNS in Chicago and the school in cleveland, 
, only five visits.could be made by the evaluation team. With 


a total budget of only $8,700, CNS trained a college student 


to. be the field researcher sather than hiring a professional. 


4 We wil] use “evaluation team" throughout the paper to 


i- describe the CNS role in- ‘the evaluation. For lack of a more 
. descriptive term, evaluation team should be read ‘to mean 
facilitators, change agents, problem solvers and helpers 


as well as outside consuitants on evaluation. 


. oe 2 ‘ i 


"5 Alternative schools often attempt to introduce many 
innovations sinuidaneousiy: New School decided the evalu- 
ation should focus on the following innovations: 1. Responsi- 
bility--the planners hoped students would come to see educa- 
tion as something they have control over and not something - — 
that will terminate when they leave the school setting. 7 
2 Involvement--planners hoped that students and teachers 
would build a community of tearners and that students would r 
‘share in running the school. 3. Learning--planners hoped 
that New School would open up possibilities for learning 
not typically available in schools. For example, use of 
‘outside resources, independent study, multidisciplinary 


courses. 


: 


6. We use school community to refer to the community life 


of the school itself as a sub-culture and not to the larger 


geographic community in which the school-is located. : 


—_ ‘ e 


Settee personnes anal detentions ha antigen ee . 
' Some critics have claimed that ethnographers have strayed 


too far in the direction of trying to conduct studies without 
any prior facuses. Ethnographic ‘studies actually can vary 

in how focused and theoretically prespecified they are-- 
varying from completely prespecified and closed, as in 
traditional experimental date, to completely open and 


» 
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atheoretical..,In the evaluation studies we have conducted, 


we have placed ourselves somewhere in the middle of this 


continuum. We have derived “inquiry focuses” from general 
inglovative program @oals and from our knowledge of similar #* 
settings. Our initial oriénting question is: what in the 
events or social systen of this program can help to explain 
either the accomp]ishment or the failure of any particular 
goal? Thus, while not totally open, we have much laitiiae: : 


in persuing relevant data. 


8 None of the evaluation studies we have conducted has been - ~- 
perfect. All those involved have experienced a mixture oft ' 


satisfaction ‘and’ dissatisfaction. It would be impossible 


oe undesirable to fix lane on particular groups, and 


undoubtedly all the principals could have performed their’ 


a P : + 
roles petter--the field observers, the evaluation team, the 


school communities. We believe, however, that no matter how 
skillful those involved in these studies might be, they 

would confront many of the problems we discuss. For the 

sake of analyis we emphasise the difficulties. this analysis 
draws heavily’ a the ca candid reflections of teacher, student and 
field worker at New School. 


ee 


9 It shoyld be noted_that the teacher and student papers 

we are quoting from the New School evaluation were written 

by members of the power clique. We have analyzed some of 

the reeshia ie this group's dissatisfaction with the field 
worker, and the dynamics of the situation should be considered 
when readive their comments., It would be impossible to deter- 
mine non much ‘their: feelings were shared by other community 
members. + * 36 
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